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PREFACE 



This report is part of the National Center for Research in Vocational Education's 
(NCRVE) continuing effort to improve vocational and technical curriculum and ^'isuuction. 
This study is one in a series of investigations being conducted by researchers at the 
NCRVE that examine how people learn technical information and how that information can 
best be taught. This particular study describes the development and testing of an intelligent 
computer program that coaches students as they troubleshoot simulated aircraft electrical 
system faults. It is hoped that this developmental study will be of interest to researchers, 
practitioners, and policymakers in vocational and technical education who are interested in 
improving the quality and effectiveness of technical training in private industry, community 
colleges, and vocational schools. 
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EXECUTIVE SUMMARY 



Technological advances create problems for those involved in keeping technical 
systems operational. To be successful with the new technologies, maintenance technicians 
must have a good understanding of technological systems and possess transferable skills 
that enable them to access new information as technologies change. Because of the 
advances in technology, the most valuable job skill of the future may be the ability to think. 

An effective educational training program for preparing troubleshooters is one that 
provides the knowledge needed to understand the technology, teaches the skills of 
troubleshooting, and provides students with the opportunity to practice using their 
knowledge and skills to diagnose faulty equipment. In the past, trainers have tended to 
overemphasize the theoretical concepts of technical systems at the expense of 
troubleshooting skill development. The tendency of instructors to emphasize theory before 
practice is partly due to the effect of prior instruction; that is, "we teach as we were taught." 
Theory-oriented instruction is also easier to plan, manage, and deliver than truly effective 
activity-based training. Instructors who desire to increase their instructional emphasis on 
experiential learning are often hampered by the lack of equipment for training purposes, 
limited availability of tooling, problems with wear and tear inherent in the process of 
assembly and disassembly, and the increased time necer>sary to physically do all of the 
activities essential for practical training. 

Fortunately, instructional technologies are capable of supporting technical 
instruction. Intelligent tutoring systems are one form of instructional technology that utilize 
advanced computer technologies to coach a student through a learning experience. These 
systems offer instructors the flexibility to conduct practical training with more efficiency 
because they are no longer constrained by limited equipment and laboratory time. Learners 
can experience realistic troubleshooting during laboratory sessions and on their own time. 
As a result, more hands-on experiences can be provided in a shorter amount of time than is 
possible through traditional laboratory practice. 

A tutoring system called the Technical Troubleshooting Tutor has recently been 
developed which simulates troubleshooting scenarios so students can practice 
troubleshooting aircraft electrical systems. The purpose of this study was to assess the 
effectiveness of the Technical Troubleshooting Tutor for developing troubleshooting skills 
in technicians. 
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Tutor Description 



The Ti?chnical Troubleshooting Tutor is a form of intelligeni tutoring system that 
can be characterized as a computer-coached practice environment. Several pedagogical 
principles identified in the cognitive science literature have been incorporated into the Tutor. 
These principles include incorporating components of apprenticeship such as coaching, 
fading, and scaffolding; providing a motivating microworld environment: utilizing real 
problems, situations, and contexts; maximizing the time spent on cognitive activity; 
reducing cognitive overload during practice; and nunuring and rewarding expert behavior. 
The Tutor provides a structured practice environment for students that is designed around 
realistic computer-displayed fault simulations. Students are presented with problem 
scenarios that they attempt to solve by collecting and interpreting information, developing a 
problem space, and selecting procedures to collect information in order to test potential 
faults. During the problem-solving activity, students are coached by the computer to think 
and perform like an expert. 



Method 

The target population for this study consisted of sophomores and juniors enrolled in 
the Institute of Aviation at the University of Illinois at Urbana-Champaign for the purpose 
of obtaining Airframe and Powerplant certification from the Federal Aviation 
Administration. The Technical Troubleshooting Tutor was incorporated into the second- 
level course. Aircraft Systems II. The control group consisted of sixteen students enrolled 
in the course during Fall semester, 1990, while the experimental group consisted of 
eighteen students enrolled during Fall semester, 1991. 

The conu-ol group subjects completed all of the requirements of the existing course 
and then participated in several troubleshooting performance tasks. In addition to 
completing the customary coursework and examinations, the tutor group subjects 
participated in the u*oubleshooting tutor treatment. This treatment involved working on the 
Tutor to practice solving aircraft electrical system faults. After completing the Tutor 
exercises, each student participated in the same troubleshooting performance tasks used 
with the control subjects. 

The troubleshooting performance task allowed for comparisons of the effect of the 
Tutor on troubleshooting ability. Each student was individually presented with an aircraft 
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electrical system simulator board in which four independent faults were inserted. Students 
were given common troubleshooting tools and were asked to locate the faults. Verbal 
protocols were collected and analyzed to identify the cognitive processes used during 
troubleshooting. Treatment effects were examined by comparing performance on the 
transfer task. The relationship between aptitude, domain knowledge, and task performance 
was also examined. In the last week of instruction, all subjects completed a domain 
specific examination which also included a demographic questionnaire. 



Treatment Characteristics 

The tutor subjects averaged five hours and fifteen minutes on the computer Tutor 
during which they solved an average of thirty problems. The computer problems took an 
average of 10.4 minutes to solve while real laboratory problems were estimated to ti^ke 
more than twenty-eight minutes to solve. Based on these estimates, the computer allowed 
for a time savings of nine hours and fourteen minutes per student which is a sixty-three 
percent time savings for simulated versus real problems. As a result, the computer allowed 
the subjects to complete many more problems in the same amount of time than they could 
have completed in a traditional laboratory. In addition, the Tutor allowed the students to 
gain considerable experience in proper use of the cognitive strategies needed for competent 
troubleshooting because it emphasized cognitive skills and de-emphasized physical skills. 



Troubleshooting Performance Differences 

Near the end of each semester, all students individually participated in a 
pertbrmance task that required them to troubleshoot a faulty aircraft electrical system in 
which four independent faults had been inserted. There were numerous interesting 
differences between the two groups on their ability to complete these real troubleshooting 
tasks. 

Ability to Recognize that Faults Exist 

There was no significant difference in the ability of the control and tutor groups to 
recognize that faults existed in the electrical system. Overall, the control group was able to 
recognize ninety-one percent of the faults while the tutor group recognized eighty-nine 
percent of the faults. 
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Ability to Locate Faults 

The most reliable indicator of troubleshooting performance is the realistic 
demonstration of that skill. In spite of the fact that both groups of subjects were equally 
able to recognize that faults existed within the aircraft electrical system, there was a highly 
significant difference in their ability to v'lctually locate and identify the faults. The control 
group subjects solved an average of 1.63 problems while the tutor group subjects solved an 
average of 2.89 problems — a seventy-eight percent improvement in trouble$;hooting 
success over the non-tutor group. As a group, the conti'ol subjects solved less than half of 
the attempted problems while the tutor group subjects solved seventy-two percent of the 
problems. 

Electrical Domain Knowledge Differences 

Due to the fact that the tutor group outperformed the control group on the 
troubleshooting performance task, one might suspect that the treatment group had learned 
more about the characteristics of electrical circuits and their components. This, however, 
was not the case. There was no difference in the mean scores on the domain-referenced 
electrical system posttest examination for the tutor and control groups. Although the Tutor 
seems to enhance the troubleshooting ability of its users, it does not appear to increase their 
declarative knowledge of electrical system structure, function, and behavior. This finding 
is consistent with the results of other studies that have examined the relationship between 
domain knowledge and performance. 

Cognitive Processing Differences 

Research suggests that experts evaluate problems qualitatively prior to taking 
action. Typically this involves careful examination of the symptoms in order to predict the 
potential fault. During problem solving, experts constantly monitor their actions to 
determine whether their predictions about the problem were correct. This monitoring 
process allows the expert to review and change strategies until the problem is solved. The 
process of thinking about one's thinking is referred to as metacognition. 

In this study, several metacognitive operations were used by both the control and 
tutor subjects. Of particular interest in this troubleshooting study are the subjects' 
metacognitive abilities to (I) predict the cause of the problem before taking action, (2) select 



appropriate troubleshooting strategies; and (3) monitor feedback for errors and sclf- 
vorrection. 

Hypothesis Selection and Evaluation 

The subjects' protocols were examined to identify the number of hypotheses they 
generated. Hypothesis generation is a major phase of the Technical Troubleshooting Model 
and serves as a goal-setting process to guide the troubleshootcr in the selection of potential 
faults. Hypothesis statements were those comments made by the subjects that suggested a 
potential cause of the problem. There was no significant difference in the ability of the two 
groups to generate plausible hypotheses. While there was no difference in their ability to 
identify potential faults, there were significant differences in their ability to correctly 
evaluate the faults. The tutor group was significantly better able to correctly evaluate their 
hypotheses than the control group. As one would expect, the tutor group was also less 
likely to incorrectly evaluate their hypotheses. Although not statistically significant, the 
control group was more likely than the tutor group to make no decision about the 
correctness of a hypothesis. These results suggest that the experience on the Tutor 
enhanced the students' abiliues to correctly evaluate the potential faults they considered. 

Troubleshooting Strategy Selection 

The type of troubleshooting strategy used certainly contributes to successful 
performance. An important difference was evident in the strategies used between the 
control and tutor groups. Tutor group m.embers were more likely to thoroughly evaluate 
the symptoms before selecting a troubleshooting strategy, they used more powerful voltage 
checks than the control subjects, and they were not dependent upon a single strategy to 
facilitate the troubleshooting process. 

Monitoring Errors and Self-Correction 

An important aspect of the troubleshooting process is whether the troubleshootcr 
realizes that an error has occurred and that self-correction is possible. This metacognitive 
operation was used to compare the types of errors committed by the two groups. 
Significant errors committed by the subjects included redundant checks, senseless checks 
made out of the problem space, and misinterpretations of acquired information. While the 
number of redundant and senseless checks made were similar for both groups, the tutor 
group tended to make fewer misinterpretation errors than the control group. The tutor 
group subjects also exhibited a stronger ability to recover from their errors than the control 
group. 

O ix 1 9 
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Student Perceptions of the Tutor 



(n addition to the collection of comparative performance data, observations of the 
students as they interacted with the Tutor were conducted to obtain formative evaluation 
data. Interviews v/ere also conducted with the tutor group students after they had 
completed the Tutor. Because it vvas in its development stage, this formative evaluation 
was crucial for future improvement of the Tutor. 

Overall, the students stated that they enjoyed working with the Tutor and said they 
would most definitely recommend it to others. The students stated that they liked the 
graphics, the user-friendliness of the program, and the fact that everything they needed for 
troubleshooting was in front of them on the screen. The majority of the students found the 
problems on the Tutor to be challenging, and they felt that it had made them better 
troubleshooters. The students also indicated that the Tutor had improved their perception 
of the electrical systems course. Some of the students reported that their prior experience in 
the prerequisite electrical course was disappointing and they, therefore, expected to have a 
similar experience in the present course. However, they found that the opportunity to solve 
a large number of simulated problems on the Tutor made them feel more positive toward 
the domain of electricity. 
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INTRODUCTION 



Technological advances create problems for those involved in keeping technical 
systems operational. The technicians and mechanics who diagnose and repair today's 
increasingly complex equipment need different skills than were needed in the past. 
Mechanics who like to work with their hands and need to see how a device works in order 
to understand it are having difficulty maintaining sophisticated equipment. These "hands- 
on" oriented individuals can no longer rely on their perceptual and physical abilities to help 
them solve technical problems. The changes in technology have reduced the extent to 
which information that is critical for detecting faults can be obtained through direct 
perception. As a result, the increasing use of electronics, small scale miniaturization, and 
the great complexity of today's equipment has led to an increase in the importance of 
abstract thinking abilities. These new technologies require mental skill above physical skill 
and are highly knowledge intensive. Successful maintenance technicians in the future will 
be those who develop troubleshooting skills that involve general as well as specific 
understanding of technological systems. They will be those who develop the transferable 
skills that enable them to access new information as technologies change. Because of the 
advances in technology, the most valuable job skill of the future may be the abiUt>' to think. 

While troubleshooting is becoming an increasingly important job skill, the 
development of good troubleshooters is a very difficult task. Troubleshooting is more than 
following a set of procedures in a service manual or practicing tasks over and over until 
they are perfected. Troubleshooting requires technicians to use their knowledge, skill, and 
experience to effectively interact with a complex technical system that is behaving in some 
unusual way. While some individuals seem to have a knack for developing 
troubleshooting skills, current research suggests that troubleshooting skills can be 
developed through properly designed instruction. For example, studies have identified the 
critical knowledge base for successful troubleshooting (Keller, 1985; Kuipers & Kassirer, 
1984; Morris & Rouse, 1986), troubleshooting strategies used by expert troubleshooters 
(Johnson, 1989; Rasmussen & Jensen, 1974), and recommended techniques for improving 
troubleshooting training (Gott, 1988; Kieras & Bovair, 1984; Lesgold et al., 1988; Morris 
& Rouse, 1986; White & Frederiksen, 1987). 

An effective training program for preparing troubleshooters is one that provides the 
knowledge needed to understand the technology, teaches the skills of troubleshooting, and 
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provides students with the opportunity to practice using their knowledge and skills to 
diagnose faulty equipment. In the past, technical trainers have tended to overemphasize the 
theoretical concepts of technical systems at the expense of troubleshooting skill 
development. The tendency of instructors to emphasize theory before practice is partly due 
to the effect of prior insuruction; that is, "we teach as we were taught." Traditional theory- 
oriented instruction is also easier to plan, manage, and deliver than the more effective 
activity-based training. Instructors who desire to increase their instructional effectiveness 
through experiential learning activities are often hampered by the lack of equipment for 
training purposes, limited availability of tooling, problems with wear and tear inherent in 
the process of assembly and disassembly, and the increased time necessary to physically do 
all of the activities essential for practical training. 

Fortunately, instructional technologies are capable of supporting technical 
instruction. Although computers have been used extensively in educational settings, most 
applications have been confined to drill and practice while little effort has been directed to 
the support of higher level thinking skills. Computers can have a tremendous impact on the 
learning process when integrated with high quality instructional software. When the power 
of instructional technologies is combined with our increased understanding of 
uroubleshooting expertise, more effective technical instruction can be developed. 

Intelligent tutoring systems (ITS) are one form of instructional technology that have 
great potential for improving technical instruction. These systems are very powerful 
computer-based instruction programs that use advanced computer technologies to 
incorporate an expert's domain knowledge to tutor or coach a student through a learning 
experience. ITS offer instructors the flexibility to conduct practical training with more 
efficiency because they reduce the constraints of limited equipment and laboratory time. 
With well designed software, learners can experience realistic troubleshooting during 
scheduled laboratory sessions and on their own time. As a result, more hands on 
experience can be provided in a shorter amount of time than is possible through traditional 
laboratory practice. 

A tutoring system called the Technical Troubleshooting Tutor has recently been 
developed which simulates troubleshooting scenarios so students can practice 
troubleshooting faulty aircraft electrical systems. The purpose of this study was to assess 
the efficacy of the Tutor for developing troubleshooting skills in maintenance technicians. 



The tbllowing research questions were developed to assess the impact of the Technical 
Troubleshooting Tutor on troubleshooting ability: 



1 . Does the computer-based Tutor improve student's ability to solve authentic 
electrical system faults? 

2. Is there a posttreatm.ent difference between the Tutor and non-tutor groups' 
declarative knowledge of electrical systems? 

3. Is there a posttreatment difference between the Tutor and non-tutor groups' use of 
metacognitive skills while troubleshooting? 

This study also examined the students' perceptions of the usefulness and 
effectiveness of the Tutor for the purpose of providing formative evaluation data. 



TUTOR DESCRIPTION 

The Technical Troubleshooting Tutor was originally designed to serve as a research 
tool rather than an instructional apparatus. When instructional strategy research is 
conducted in authentic classroom settings, however, concerns arise about the influence of 
the instructor on the success or failure of the strategy. The success of a strategy may be 
more attributable to the quality of the instructor than the strategy itself. To reduce the 
influence of instructor quality in this study, the Tutor was designed to control the delivery 
of instruction so that all students would receive the same quality of instruction. This design 
allowed the assessment of instructional design effects without instructor quality interaction. 

The Technical Troubleshooting Tutor can be characterized as a computer-coached 
practice environment for cognitive enhancement (Lajoie & Lesgoid, 1989). As a cognitive 
enhancer, the tutor is designed to help students who possess a set of prerequisite domain 
knowledge and skills. The program provides a microworld practice environment designed 
around realistic computer-generated fault simulations. Students are presented with problem 
scenarios that they attempt to solve by collecting and interpreting information, developing a 
problem space, and selecting procedures to collect information in order to test potential 
faults. During the problem-solving activity, students are coached by the computer to think 
and perform like an expert. 
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In *adiiional instructional settings, it is common for students to solve only a limited 
number of technical problems in a semester. This is due to a combination of factors. First, 
troubleshooting exercises take a long time to complete because oi the equipment 
manipulations and technical tests that must be done. Second, few school laboratories have 
sufficient work stations to allow an entire class to engage in troubleshooting exercises at 
one time. Without sufficient numbers of work stations, insuuctors must be creative in their 
selection of activities so students remain busy during class time. As a result, considerable 
laboratory time is spent on tasks that are ancillary to actual troubleshooting experiences 
(e.g., soldering, crimping, and circuit design exercises). Because the Tutor emphasizes the 
cognitive activity involved in troubleshooting and de-emphasizes time-consuming physical 
manipulations, students are provided with the opportunity to solve many technical 
problems in a short amount of time. In addition, by simply adding a computer or two to a 
laboratory environment, additional work stations are provided. As a result of these 
improvements, extensive structured practice opportunities can be provided for students 
which help them quickly develop the same mental patterns that are developed by expert 
problem solvers through many years of troubleshooting experience (Nichols, Pokorny, 
Jones, Gott, & Alley, 1989). 



Pedagogical Principles of the Tutor 

The Technical Troubleshooting Tutor was designed around the Technical 
Troubleshooting Model and graphical problem space concept developed at the Training and 
Development Research Center at the University of Minnesota (Johnson, 1987). Several 
pedagogical principles identified in the cognitive science literature have bcc^n incorporated 
into the Tutor. These principles include incorporating components of cognitive 
apprenticeship such as coaching, fading, and scaffolding; providing a motivating 
microworld environment; utilizing real problems, situations, and contexts; maximizing the 
tin-e spent on cognitive activity; reducing cognitive overload during practice; and nurturing 
and rewarding expert behavior. 

Incorporate the Cognitive Apprenticeship Model of Instruction 

Through the years, vocational and technical instructors have utilized various forms 
of apprenticeship in their instruction. Traditional apprenticeship typically involves an 
expert who models the desired quality of performance for novices, coaches them through a 



task, and gives ihem more autonomy as their skills develop. In a traditional craft guild, for 
example, the master models expert behavior by demonsu-ating to the apprentice how to do a 
task while explaining what is being done and why it is done that way. By observing the 
master perform, the apprentice learns the correct actions and procedures and then attempts 
to copy them on a similar task. The master then coaches the apprentice through the task by 
providing hints and corrective feedback as needed. As the apprentice becomes more 
skilled, the master gives the apprentice more control over tae task by "fading" into the 
background. 

Coach Students Through Difficult Situations 

While traditional apprenticeship emphasizes physical ability, a modification of this 
approach called cognitive apprenticeship has been adapted for the Tutor to enhance student 
cognitive abilities. The Tutor records student actions during troubleshooting, analyzes their 
performance, and provides hints and assistance (see Figure 1 for an example of a coaching 
explanation). Proper techniques for collecting information, representing the problem 
space, and selecting and performing technical tests are nurtured by the Tutor through 
explicit support and guidance when they are most needed. 

Sequence Learning Experiences Based on Individual Performance 

The effectiveness of cognitive apprenticeship can be further enhanced by 
individualizing the selection of the problems faced by students. Just as effective teachers 
pay close attention to each student's current level of ability and give them tasks that build 
on their prior learning, the Tutor sequences learning experiences based on individual 
student performance. After each problem is solved by the student, the Tutor assesses the 
student's performance by compiling the data collected on twenty-five troubleshooting 
performance indicators and makes norm-referenced comparisons to other students' 
performances. Based on this comparison, the Tutor makes a decision to give the student an 
easier or more difficult problem to attempt next. The Tutor also determines if the student 
has successfully reached a criterion level of performance on each problem type (i.e., 
component failure, open circuit, short circuit). For example, if a student has successfully 
solved five "open circuit" problems at a high level of performance, the Tutor will present 
the student with problems that involve "short circuits." 




Fade into the Background as Student Ability Improves 

As students' thinking processes develop, they are able to perform with little 
instructor intervention. This fading aspect of cognitive apprenticeship results in the gradual 
transfer of responsibility for learning from teacher to student. As students progress 
through the Tutor's problem type and difficulty hierarchy, they are provided with less 
problem-specific information and reduced levels of coaching. This fading aspect of 
instruction forces students to rely more on their own knowledge and skills and less on their 
interactions with the Tutor. 

Help Students Develop Automaticity Through Practice 

One of the key characteristics of expertise is an apparent ease of performance. 
Experts are able to perform quickly, fluently, and efficiently. There is general agreement 
that practice is essential for the development of skilled performance. Research indicates 
that practice leads to an increase in the speed of task completion and a decrease in error rate 
(Phye, 1986). 

Practice usually involves repetition of a task or skill. Cognitive science researchers 
have identified the following conditions needed for practice to bring about speed and 
accuracy improvements: (1) knowledge of results, (2) causal attribution, (3) generation of 
alternatives, (4) hindsight, and (5) learning from instruction (Langley & Simon, 1981). 

During practice, students must receive feedback about the results of their actions 
before they can improve. Knowledge of results allows students to monitor their problem- 
solving performance by providing information about the correcmess of their performance, 
the length of time each problem took to solve, and the types and number of errors they 
made. 

Through practice, students begin to see relationships between actions, conditions, 
and outcomes. For example, students in an electronics course may learn that closing a 
switch will cause a relay to energize only if a power source is available. The students may 
also learn that if the power source is weak or if the switch is faulty, the relay will not be 
energized. This awareness of causal attribution enables students to generate alternative 
solutions to problems. Using the above example, a student who recognizes the causal 
relationships between the switch, power source, and relay can identify alternative fault 
possibilities for a problem in which the relay will not energize. The student who does not 



understand the causal relationship between the three components will have greater difficulty 
developing plausible solutions to a problem. 

Hindsight, the fourth condiuon of effective practice, involves the examination and 
evaluation of past performance based on knowledge of results and causal attribution (Phye, 
1986). Through practice, students build a repertoire of successes and failures. By using 
hindsight to examine past performances, students can recognize patterns in their successes 
and failures. They can begin to recognize their own mistakes and the misconceptions thai 
caused the errors. In this way, hindsight can improve future performance. 

All four of these conditions are necessary to bring about increased speed, decreased 
errors, and automaticity of skill. Using the Tutor to provide opportunities for practice 
should facilitate and augment learning by providing feedback, developing causal 
understanding, emphasizing the generation of potential faults, and encouraging the use of 
hindsight to examine performance. 

Provide a Motivating Microworid Environment 

Intrinsic motivation is an essential factor in learning (Bruner, 1962). When 
learning activities are intrinsically motivating to students, they may spend more time on the 
activity, put more effort into learning, feel better about what they have learned, and be more 
likely to use their new knowledge and skill in the future (Malone, 1981). Based on a series 
of interviews with highly motivated people, Csikszentmihalyi (1978) identified the 
following features of intrinsically motivating activities: 

1 . The level of challenge for students increases or decreases based on their current . 
level of skill. 

2. Important instructional activities are isolated from other events or activities which 
may interfere, 

3. There is a criteria for performance that clearly informs the student of his or her 
current progress toward the criterion. 

4. Concrete feedback is provided to the student 

5 . The student is confronted with a broad range of challenges. 



The Tutor was designed to promote intrinsic motivation through the incorporation 
of Csikszentmihalyi's features of intrinsically motivating activities. For example, upon 
completion of each problem, the Tutor evaluates the student's performance and chooses the 
next problem based on that performance (see Figure 2). Students are also given 
considerable feedback while solving the problems; comparisons of their performance to that 
of other students are made; and a very broad range of problem types covering an entire 
aircraft elecuical system have been built into the Tutor. 

Intrinsic motivation can also be enhanced through activities that embody fantasy and 
curiosity (Malone, 1981). The Tutor encourages fantasy and curiosity by simulating real 
activity on the computer. Exam.ples of the variety of the problem scenarios that promote 
fantasy and curiosity include an aircraft that has been confiscated from drug smugglers, 
problems that occurred while student pilots were using the aircraft, the salvage operation of 
a wrecked aircraft, and troubleshooting problems encountered during a competition 
between aviation students from two universities. 

As students work on each troubleshooting scenario, they need to perform a variety 
of checks which cost money and time. Using time and labor figures derived from actual 
aircraft maintenance settings, the Tutor records the time and cost required for each 
procedure used by students and provides them with an updated record of their overall cost 
and time performance (see Figure 3). The student's goal is to keep the simulated costs as 
low as possible and the simulated work time as short as possible. 



Figure 2 

Evaluating Student Troubleshooting Performance 
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Figure 3 

Running Total of Work Time and Labor Costs for this Problem 
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Utilize Real Problems, Situations, and Contexts 

Instruction often promotes an understanding that the teacher is the all knowing 
authority, that problems are simple and straightforward, that problems can be solved by 
applying the methods or formulas just covered in class, and that there is usually only one 
right answer to a problem. In contrast, instruction that occurs in real world contexts 
promotes a much different type of understanding. Real world instruction promotes an 
understanding that the nature of knowledge is uncertain, that learning is not a totally orderly 
process, and that not all problems have straightforward and simple solutions. Knowledge 
gained through realistic activity is, thus, more likely to be used in future situations. 

Cognitive research indicates that people learn because of the contextual informadon 
in the problem situation. Situated learning is a term that describes the acquisition of 
knowledge and skills in an instructional context that reflects the way the knowledge and 
skills will be used in real life (Brown, Collins, & Duguid, 1988). This concept is not new 
to education. Dewey (1956) urged basing education in reality and suggested that each day 
a student should bring home from school something which could be used that day. 

Research has consistently indicated that the way something is learned influences 
later use of that knowledge. It appears that knowledge is indexed when it is learned so that 
it can be found and retrieved when needed at a later time (Glass, Holyoak, & Santa, 1979; 
Phye, 1986; Reiser, 1986). Several researchers have pointed out the importance of context 
for indexing knowledge to be stored in memory. For example, problem-oriented 
instruction too often takes place in contexts that are dissimilar from those the student will 
encounter later. Providing the opportunity for students to process information in a 
problem-oriented format appears to help them acquire conditionalized knowledge— 
knowledge that includes information about the conditions and constraints of its use 
(Anderson, 1983; Bereiter, 1984; Glaser, 1984). Students who learn under such 
conditions will be more likely to spontaneously use their new knowledge when necessary. 
Consequently, careful selection and planning of the instructional context is of p^^ne 
importance in instructional design. 

Learning within real world contexts does not mean that instruction must take place 
outside the school classroom to be effective. The Tutor was designed to use problems, 
situations, and contexts that the students would face as technicians in the aircraft 
maintenance industry. These experiential learning activities present students with real 
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world problems and challenge ihem lo collect and interpret the available symptoms, develop 
a set of potential faults, and derive methods for testing those potential faults. Figure 4 
provides one example of the realistic problem information used in the Tutor. 
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Figure 4 

Example of Realistic Problem Information 
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Maximize the Time Spent on Cognitive Activity 

One of the keys to the development of expertise is having extensive opportunities to 
practice using cognitive skills (Phye, 1986). The cognitive skills needed for 
troubleshooting include skills such as acquiring and interpreting information, identifying 
attributes and components, recognizing patterns, and generating and evaluating hypotheses. 
In most technical courses, however, instruction takes the form of teacher-directed lectures 
and physical skill development activities in a laboratory (Johnson, 1990). Students tend to 
be passive receptors of information during lectures and often spend their laboratory time 
replicating activities that have already been performed by the instructor. For example, 
students will often use laboratory time to complete non-troubleshooting related tasks such 
as soldering, crimping various types of wire connectors, splicing wires, and developing 
basic hand tool proficiency. As a result, the amount of time students actually spend on 
cognitive activity is very limited. 

Building on the current capabilities of computer technology, high fidelity 
troubleshooting scenarios have been designed into the Tutor (see Figure 5 for an example 
of the digital meter that is simulated in the Tutor). Real troubleshooting tends to take a long 
time due to the physical actions needed to collect information mn tests, and make repairs. 
Because students do not have to physically remove panels ana . jsen bolts, they can solve 
many problems on the computer faster than they could on real equipment. It is expected 
that reducing the time required to solve each problem will allow students to solve many 
problems and, therefore, gain more relevant troubleshooting experience in a shorter amount 
of time. 



33 

15 



Figure 5 

Digital Meter as Simulated by the Computer 
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Providing troubleshooting experience on the computer also improves laboratory 
management for instructors and provides individualized instruction for students. Technical 
instructors are often hampered by the lack of equipment for training purposes, limited 
availability of tooling, and problems of wear and tear on the training equipment due to the 
constant assembly and disassembly activities. These limitations can result in only a few 
students actually doing troubleshooting while the rest of a class completes non-essential, 
"busy work" activities in the lab. With troubleshooting training software available on 
computers, instructors can increase the number of work stations in a laboratory to keep a 
greater percentage of the class "on-task," This results in an increase in the amount of direct 
instruction received by each student. By having more students actually engaged in 
troubleshooting activity, the amount of time spent on cognitive activity is increased. 

Reduce Cognitive Overload During Practice 

Research shows that expens are able to process a large amount of information when 
solving problems while novices often get "mentally bogged down." Instruction needs to 
help students reduce the overload on their working memory in order to enhance their ability 
to learn and solve problems. One way to reduce the load on working memory is through 
the use of an external memory. External memories can be as simple as a list of things to do 
or as complicated as a diagram of an electronic device. An external memory reduces 
working memory load in the following three ways; (1) It contains information that does 
not need to be retained in mem.ory; (2) it allows manipulation of information outside of 
working memory; and (3) it provides a visual, perceptual, and accessible record of a 
sequence or process that otherwise would need to be kept in working memory. External 
memory also enables problem solvers to keep track of where they are in the process of 
solving a problem, thereby easing the load on working memory (Larkin, 1988). 

Several forms of external memories have been designed into the Tutor to help 
reduce cognitive overload during the troubleshooting process. These include the use of 
concept maps, the availability of troubleshooting performance records, and focusing 
attention through the development of a graphical problem, space. 

Concept Maps 

Concept maps are a form of external memory aid that help students organize new 
information. Concept maps were originally developed as a tool for researchers to help 



ihem learn how people organize knowledge (Novak, Gowin, &. Johansen, 1983). They 
are now being used as instructional tools to help learners process information for learning 
and to evaluate student learning. Concept maps help students distinguish important 
concepts, arrange concepts in some meaningful order, and eotablish significant 
relationships between concepts. The ability of concept maps to organize information makes 
it easier for students to form "chunks" of concepts that can be stored in memory. 

The concept maps that have been integrated into the Tutor are called functional flow 
diagrams and are designed to help students organize their understanding of the aircraft 
electrical system (see Figure 6). Functional flow diagrams differ from the schematic 
diagrams that are commonly used for technical instruction and troubleshooting. The 
following list identifies the major differences between functional flow diagrams and 
schematic diagrams: 

1 . Functional flow diagrams present a simplistic view of the system, displaying only 
the system's essential component parts, while schematic diagrams display all 
component parts within the system. As a result, students who learn from concept 
maps will initially gain an understanding of the "big picture" without all the detail. 

2. Functional flow diagrams can convey causal relationships between the system s 
essential component parts (e.g., activating component A causes component B to 
activate), while schematic diagrams do not explicitly convey causal relationships. 

3. Functional flow diagrams imply a time sequence within the system (i.e., component 
A must change before component B changes), while schematic diagrams represent 
the system at only one point in time. 

4. The functional flow diagrams can explicitly display the motion of flow through the 
system by the use of arrows and action-oriented concept labels, while schematic 
diagrams typically display the system in a uonary or static state. 

5. Functional flow diagrams reinforce a critical systems view by explicitly showing 
common systems and subsystems. While schematic diagrams show subsystem 
circuits, they are not readily evident to individuals who lack a general understanding 
of the entire system (Johnson & Satchwell, 1992). 
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Figure 6 

Functionai Flow Diagram of a Small Aircraft's Electrical System 
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The Tutor also provides an external memory for students by keeping records of 
student performance for a wide variety of performance indicators (see Figure 7). Many of 
the pertbrmance indicators are available to the student for review during the troubleshooting 
process. For example, the student is able to look at a list of ihe potential faults that were 
considered, the types of information that were acquired, and the technical tests that were 
run. Having this information available from the computer allows the troubleshooter to free 
up working memory and concentrate on only the most imponant information cues. 

When working on complex problems, many troubleshooters selectively reduce the 
size of the problem space by eliminating potential faults through the use of various technical 
tests (Johnson, 1988). This reduction in problem space size serves to decrease the amount 
of information that must be attended to in working memory. The Tutor also allows 
students to reduce the size of the problem space as they attempt solutions. Students are 
able to delete potential faults from a master list and are shown a visual representation of the 
reduced problem space on a concept map. 

Strategically Focus Learner Attention 

Working memory processes what the senses take in. Because the senses are 
continually flooded with information and attentionai resources are limited, individuals must 
be able to control what information gets irto their working memory. The human attentionai 
system is used to prevent memory overload by ensuring that only the information which is 
"attended to" will be put into working memory. 

Attentionai focus has been described as a prerequisite for learning (Grabe, 1986). 
One way of easing the load on working memory is to guide learners to direct their attention 
towards the most critical information so that it can be encoded. According to Kulhavy, 
Peterson, and Schwartz (1986), any procedure that directs attention to the instructional 
content increases the probability that learners will learn the intended material. 

Experts have been found to notice relevant and subtle features of events that are not 
recognized by novices. An important goal in the process of facilitating development toward 
expertise is helping novices notice relevant features of problems. Novices need to become 
sensitive to features and dimensions tliat otherwise might escape their attention. 
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Figure 7 

Troubleshooting Performance Report 
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Grabe (1986) describes a number of instructional strategies for guiding learners 
attentional focus during the instructional process. The first is a scanning strategy in which 
learners identify problem components, compare a target problem to a set of alternative 
features for each component, and then eliminate those alternatives that do not match. A 
second strategy involves devising ways to help learners identify important sources ot 
information and to avoid distracting, irrelevant information. A third strategy uses 
highlighting of attentional targets with bright colors, loud sounds, and novelty. 

The functional flow diagrams used in the Tutor were designed to ease the load on 
student's working memory and to direct their attention to the key aspects of the problem. 
Based on a concept called a graphical problem space, the Tutor uses visual cues to reinforce 
the development of a problem space (see Figure 8). The student uses a mouse to point to a 
location on the screen image of the functional flow diagram where the fault may exist. 
When the student clicks the mouse button at that location, the computer will highlight that 
portion of the diagram if it could contain the fault. As the student selects more potential 
fault locations, a visual representation of the problem space is developed. If the student 
selects a location that could not contain the fault, the Tutor provides immediate feedback 
about the wrong selection. The Tutor also allows the students to query for a more elaborate 
explanation of why that location could not contain the fault. 



Figure 8 




Nurture and Reward Expert Behavior 

A synthesis of problem-solving research studies (Bouwman, 1983; Elstein. 
Shulman. & Sprafka, 1978; Sweller & Levine, 1982) resulted in the development of the 
Technical Troubleshooting Model (TTM) thai accurately reflects the cognitive process flow 
of the troubleshooter when working on a technical problem (Johnson, 1989). This model, 
as shown in Figure 9, is divided into two main phases: (1) hypothesis generation and (2) 
hypothesis evaluation. In phase one, the problem solver represents the problem by 
obtaining information from internal or external sources (see Figure 10). As an internal 
source, the individual's long-term memory contains both declarative and procedural 
knowledge (Anderson, 1980, 1982; Glaser, 1984). The external sources could include job 
aids; technical support; technical evaluations through test procedures and operational 
adjustments; and sensory-based evaluations including visual, auditory, olfactory, and 
tactile checks. In the second step of the process, the troubleshooter applies cognitive action 
to interpret the acquired information and determine its relevance in relation to the problem. 
Once the problem solver has collected sufficient information, one or more hypotheses may 
be generated (Elstein et al., 1978; Frederiksen, 1984; Johnson, 1987). 

In the second phase of this model, the troubleshooter selects a hypothesis to 
evaluate. This involves using a variety of search strategies to obtain additional information 
to support a decision to either accept or reject the proposed hypothesis (Frederiksen, 1984). 
The selection of these strategies depends on a variety of factors including the 
troubleshooter's level of expertise, the type of technical system, and the difficulty of the 
problem. The following list identifies four troubleshooting strategies that are commonly 
used by technical troubleshooters: 

1 . Exhaustive search srraregy— This strategy involves testing all fault possibilities. 
This method requires very little expertise but is only feasible if the set of possible 
faults is small. The repair of an old television with vacuum tubes is a good example 
of the appropriate use of this strategy. Rather than use more sophisticated 
troubleshooting strategies, a practical solution for fixing the television is to 
systematically test all the tubes. 

2 . Topographic search strategy— This strategy is similar to using a road map to plan a 
trip. A topographic search strategy starts at some point in the system and relies on a 
schematic to trace through the system to locate and test the components. 
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3 , Half/split search strategy — This meihod aitempis lo eliminaie ihc greatest number of 
fault possibilities with each test. The half/split strategy involves making a test at the 
midpoint of the system which reduces the search space to only one-half of the 

• system. The next check will then be at the midpoint of the remaining half of the 
circuit. 

4, Functional search strategy — Because this method requires extensive system 
knowledge and thinking ability, it is the most powerful and most difficult 
troubleshooting method. As a result, it is usually used when the above strategies 
have failed. This method involves observing the function of the system and 
proceeding to a specific subsystem based on that information. This method often 
involves mental simulation of the system in both normal and malfunctioning states. 
Information about the system and its components is collected and hypotheses about 
the fault arc formed and tested. 

While most troubleshooters rely on one or tv/o favorite strategies, each strategy is 
useful under certain circumstances. Topographic, exhaustive, and trial and error searches 
are selected because little cognitive effort is needed. Some methods such as the half/split 
are selected because they are efficient at eHminating a lai je number of possibihties. Other 
reasons for selecting methods include their ease of use, their low cost in terms of time and 
materials, and their reliance on the availability of :;pare parts and other resources. 

By using these strategies, the troubleshooter will eventually reach a decision point 
in the U'oubleshooting process. If the acquired information, and subsequent interpretation, 
confirms the selected hypothesis, the troubleshooting process ends. If the activity does not 
result in support of the selected hypothesis the troubleshooter will cycle back to phase one 
of the model and generate another hypothesis or acquire additional information that can 
contribute to the selection of a more plausible hypothesis (Johnson, 1987). 

The expert approach to troubleshooting as described above provided the framework 
for the design of the Tutor. Although the students may attempt to solve the problems in 
any manner they choose, the Tutor encourages them to follow the TTM framework by 
rewarding them with higher performance scores when they do the things expert 
troubleshooters tend to do. Rather than begin their problem-solving activity by running 
equipment checks and procedures, the students are encouraged to begin by collecting 



Figure 9 
Technical Troublesiiooting Model 
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Figure 10 

Sources of Information for Troubleshooting 
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information from a variety of sources and then develop a complete problem space based on 
the symptoms. This type of cognitive activity early in the problem-solving process 
encourages the students to reason qualitatively before taking action. This approach is 
supported by research that has investigated the differences in the ways experts and novices 
approach problems (Chi, Feltovich, & Glaser, 1981; deKleer, 1985; Larkin, McDermott, 
Simon, & Simon, 1980). When confronted with a problem, novices tend to immediately 
look for a solution to the problem. For example, when solving word problems in 
mathematics and electronic domains, novices typically look for some type of formula that 
can be applied to the information in the problem. In contrast, experts begin problems by 
analyzing the problem information from a qualitative perspective. This approach allows the 
problem solver to gain an understanding of the problem before a solution strategy is 
selected. Only when the problem is understood does the expert begin looking for a 
solution. 



METHOD 

This study examined the effect of a computer-based tutoring program on technical 
troubleshooting ability. Through a case-based microworld environment, subjects practiced 
troubleshooting by locating faults in a generic aircraft electrical system. Following the 
completion of the tutoring program, subjects participated in a set of laboratory problems 
that served as a transfer of learning task. This transfer of learning task was used to 
examine the impact of the Tutor on authentic troubleshooting performance. 



Subjects 

The target population for this study consisted of students enrolled in a second-level 
electronic systems course in the Institute of Aviation at the University of Illinois at Urbana- 
Champaign. The students enrolled in this course were university sophomores and juniors 
who were working toward Airframe and Powerplant certification from the Federal Aviation 
Administration. Due to the small enrollments in this class and the fact that it was offered 
only once each year, it was necessary to use intact classes. The control group consisted of 
sixteen students enrolled in the course during Fall semester, 1990, while the experimental 
group consisted of eighteen students enrolled during Fall semester, 1991. Because data 
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was being compared across two separate semesters, very little interaction occurred between 
the researchers and the course instructor to assure that the course would be taught in the 
same way each semester. 



The students who panicipated in this study had successfully completed prerequisite 
courses in aircraft electrical systems and powerplani systems. The foundation electronics 
course provided students v/ith basic knowledge of electrical concepts such as AC and DC 
theory, power generation, circuitry, and solid state devices. The powerplani course 
covered the theories and operating principles of ignition, starting, and electrical power 
generating components and systems found in aircraft turbine and reciprocating 
powerplants. Through these courses, students also acquired limited skills in the use of 
hand tools and common diagnostic tooling. While these prerequisite courses provided 
foundation knowledge and introductory skills, they appeared to be adequate preparation for 
the concepts and skills developed through this study. 

Because random selection of subjects was not possible, a variety of demographic 
and aptitude comparisons were made to determine if the control and treatment groups were 
similar. Demographic data was collected through a questionnaire administered at the end of 
the course. There was no significant difference in the mean ages of the two groups, r(32) = 
-1.701, /7>.05. The mean age of the 1990 subjects was 20.67 with a range of 19-28, 
while the mean age of the 1991 subjects was 22.94 with a range of 20-38. With the 
exception of one subject who was enrolled in the prerequisite course concurrently, all 
subjects had successfully completed the prerequisite aircraft electrical systems course. The 
subjects reported having a variety of related experiences including military electronics 
training, high school electronics instruction, and hobby interests, although there was no 
apparent difference between the groups on these experience variables. 

Aptitude indicators for the two groups were obtained from the archival records of 
the Institute of Aviation. These included American College Testing (ACT) Program 
examination scores. Survey of Mechanical Insight scores. University of lUinois grade point 
averages, high school class rank, and grades earned in the prerequisite basic electronics 
course. As shown in Table 1, no significant group differences in aptitude or achievement 
were identified. 
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Table 1 

Aptitude Indicators for Tutor and Control Group Subjects 



Conirol Group Tutor Group 



Aptitude Indicator 


/; 


M 


SD 


n 


M 


SD 




ACT Scores 


16 


24.06 


4.64 


17 


23.65 


4.11 


.273 


Mechanical Aptitude Scores 


16 


50.06 


29.72 


12 


57.92 


26.79 


-.721 


University GPA 


16 


3.92 


.44 


18 


3.86 


.38 


.449 


High School Rank 


16 


70.94 


20.37 


18 


71.11 


25.10 


-.022 


Prerequisite Course Grade 


15 


3.40 


.91 


18 


3.56 


.98 


-.560 



Note: * All aptitude indicator comparisons are non-significant at p>. 10. 

Procedure 

The Technical Troubleshooting Tutor was incorporated into the second-levei 
course. Aircraft Systems II, which is offered through the Institute of Aviation at the 
University of Illinois at Urbana-Champaign. This course consists of three hours of lecture 
and four hours of laboratory activities each week. Tlie control group subjects completed all 
of the requirements of the existing course and then participated in several troubleshooting 
performance tasks. In addition to completing the customary coursework and examinations, 
the tutor group subjects participated in the troubleshooting tutor treatment. This treatment 
involved working on the Tutor to practice solving aircraft electrical system faults. Two 
Macintosh Ilsi computers containing the troubleshooting Tutor software were placed in a 
computer laboratory adjoining the Aircraft Maintenance Technology department's library. 
The students were able to work on the Tutor anytime during the day between 8:00 am and 
5:00 pm. Prior to the start of the treatment, a one-hour demonstration and explanation of 
the Tutor was provided for the tutor group subjects. A graduate research assistant served 
as a supervisor whenever students were working on the Tutor. The supervisor's role was 
to answer any questions related to the operation of the Tutor but not to provide any 
assistance related to the solution of problems on the Tutor. The supervisor also collected 
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observation data related to the students' behaviors and verbalizations while working on the 
Tutor. 

After completing the Tutor exercises, each student participated in the same 
troubleshooting performance tasks used with the 1990 control group class. The 
troubleshooting performance task allowed for comparisons of the effect of the Tutor on 
troubleshooting ability. To maximize the instructional effects of the course, the 
troubleshooting transfer task was conducted in the last four weeks of the sixteen-week 
semester. Near the end of the semester, each student was individually presented with a 
simulator board that contained an aircraft electrical system in which four independent faults 
were inserted. The students were not told how many faults existed in the system, only that 
there were multiple faults inserted by the researcher. Students were given common 
troubleshooting tools and were asked to locate the faults. Verbal protocols were collected 
and analyzed to identify the cognitive processes used during troubleshooting. Treatment 
effects were examined by comparing performance on the transfer task. The relationship 
between aptitude, domain knowledge, and task performance was also examined. In the last 
week of instruction, all subjects completed a domain-referenced knowledge examination 
which also included a demographic questionnaire. 

Transfer Task Selection 

The apparatus used to determine posttreatment troubleshooting performance was an 
instructor-developed training board that represents ten discrete subsystems found in a small 
aircraft's electrical system. Aircraft components such as circuit oreakers, switches, relays, 
terminal strips, conductors, and other major functional system components (e.g., rotating 
beacon, power inverter, blower motor, fuel pump, various lights, control motors, and 
valves) are mounted on a tabletop board. System power is provided by an auxiliary power 
unit that also serves other training boards. 

The performance tasks were based on specific criteria to ensure representation of 
certain populations of tasks. A task analysis was used to identify problems that were 
commonly encountered by maintenance technicians in the aircraft industry. The transfer 
task problems were carefully selected to ensure that they were consistent with the 
knowledge and skills expectations of the course. This was to ensure that the control group 
would be given instruction and practice during the course that was similar to, but not 
identical to, the transfer tasks. The four problems that were ultimately selected for the 
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transfer task were then compared to the faults simulated in the Tutor. This was to ensure 
that they were not part of the fault set within the Tutor and, therefore, would not be solved 
by the tutor group prior to the final evaluation. 

Technical problems can be categorized as structural, functional, and behavioral 
faults (deKleer & Brown, 1983). Although the categorizations developed by deKleer and 
Brown were modeled, the specific interpretation of those terms is unique to this study. In 
this study, the terms were intended to represent specific conditions which provide 
recognizable symptom sets based on the relationship of system components and potential 
behaviors. Structural problems are the result of architectural faults including inappropriate 
or nonexistent connections. Functional problems are those that occur when a system 
component completely fails, rendering it without function. Behavioral problems occur in 
system components that present symptoms out of the normal operating range, or as a result 
of interaction between marginal components. These categorizations are not mutually 
exclusive and the symptomatic conditions that exist as a result of any particular fault could 
involve any or all of the categories. However, this level of definition does represent 
distinct populations from which representative tasks can be selected. 

Task selection was limited to problems that were not likely to result in immediate 
solution based on experience or cursory observation. Additionally, each problem was 
inserted so that it could be specifically identified. In an effort to prevent instructor bias, the 
transfer task problems were not revealed to the course instructor. Expert validation of the 
selected faults was accomplished through a review by senior electronic service technicians 
in the Department of Electrical and Computer Engineering at the University of Illinois. 

Four faults were selected for the transfer task which represented a range of 
difficulty and type. The first fault was a simulated open (i.e., concealed piece of 
transparent tape) in the point contact of the power path to the lamps within the rotating 
beacon. Although the actual fault was a structural malfunction, the simulated effect was a 
functional failure of the point contact. The second fault was a misplaced conductor on a 
relay in the blower motor circuit. This was a structural fault in the most complex 
subsystem. The third fault was an internally open instrument light switch. This functional 
fault was not observable from visual inspection. The final fault was an incorrectly wired 
microswitch in the landing gear indicator circuit. The microswitch terminals provide either 
normally open or normally closed options and the incorrect selection v/as made. Both the 
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rotating beacon and landing gear indicator faults included circuitry not presented on the 
schematic diagram which required subjects to either demonstrate an advanced level of 
component knowledge or generate new information through the troubleshooting process. 



Data Collection and Analysis 

Three kinds of data were obtained to determine the effect of the treatment: (1) 
domain-referenced test scores; (2) verbal protocols from the troubleshooting performance 
task; and (3) descriptive data based on observations, surveys, and archival records. The 
domain-referenced test was designed to assess subjects' knowledge of the structural, 
functional, and behavioral aspects of the specific system represented in the performance 
task. A total of twenty-one multiple choice questions and three schematic-referenced items 
included thirteen functional items, eight structural items, and three behavioral items. Item 
analysis was then performed to identify content and item weaknesses. Although a 
reliability correlation was performed (r = .40), reliability measures are designed for norm- 
referenced instruments and are not appropriate indicators for criterion-based instruments 
(Gronlund, 1985). The item analysis reliability indicator (KR-20) provides little indication 
of consistency within a criterion-referenced measure. A more appropriate method for 
determining consistency within criterion-based measures is objective analysis. The 
examination was reviewed by project and senior staff electronics technicians in the 
Department of Electrical and Computer Engineering at the University of Illinois for content 
validity. The domain-referenced test was administered in the last week of classes after 
completion of the troubleshooting transfer task exercise. 

Verbal protocols were collected during the troubleshooting performance task and 
analyzed to determine general troubleshooting performance such as problem recognition 
and solution accuracy. The recorded verbalizations were coded in accordance with 
methods established by Johnson (1989). A second rater coded approximately twenty 
percent of the total protocol data to validate the coding process. The coding of the two 
raters were compared for consistency with a resultant agreement coefficient above .90. 

Surveys, observations, and archival records were also used to collect ancillary data 
to 2isstss troubleshooting performance and to determine how the students felt about their 
interaction with the Tutor. A reaction questionnaire was administered after each student 
completed a few problems on the Tutor during their first session. TTie researcher designed 
reaction questionnaire and the interview guide were pilot tested with graduate students who 
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had taken part in the trial runs of the Tutor. An open-ended observation format was 
selected which required each supervisor on duty to make a written record of observable 
events occurring during the session. 

A personal interview was also conducted with each student after they completed the 
Tutor. Each interview took an average of about thirty minutes. Subjects were asked 
general questions about their opinions of the Tutor, prior experience with computers, career 
aspirations, and views of how the Tutor might have helped them become better 
troubleshooters. Specific questions related to the difficulty of the troubleshooting 
exercises, the adequacy of the feedback provided at the end of each problem, the 
availability of assistance from the computer, and the correlation of the Tutor activities to the 
course content. 

Data from the reaction questionnaire was analyzed by calculating response 
percentages of each group. A key word analysis was used to compile the interview data 
followed by frequency counts and percentage comparisons. The observation data was 
analyzed by noting those events which occurred most frequently. 



RESULTS 

The purpose of this investigation was to examine the impact of the Technical 
Troubleshooting Tutor on troubleshooting ability. Following a brief description of the 
treatment characteristics, the results are organized around the research questions by 
comparing the observed differences between the control and tutor groups in overall 
troubleshooting ability, electrical domain knowledge, and cognitive processing. Students' 
perceptio'js of the quality, effectiveness, and usefulness of the Tutor are also discussed. 

Treatment Characteristics 

The eighteen students enrolled in AVI 170, Electrical Systems II, during the 1991 
Fall semester received the computer treatment. Two computer stations were available for 
the project which meant that only two students could work at any one time. This treatment 
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group was expected to complete the Tutor in about six weeks but actually took nine weeks, 
with each subject completing the Tutor project at theii own pace. 

The tutor subjects averaged five hours and fifteen minutes on the Tutor during 
which they solved an average of thirty problems. The computer problems took an average 
of 10.4 minutes to solve while real laboratory problems are estimated to take more than 
twenty-eight minutes to solve. The computer allowed for a time savings of nine hours and 
fourteen minutes per student which calculates to a sixty-three percent time savings for 
simulated versus real problems. As a result, the computer allowed the subjects to complete 
many more problems in the same amount of time than they could have completed in the 
laboratory. A potential criticism of this type of research is that any improvements in 
performance could be attributed to the fact that the treatment group spent more time on task 
than the control group. While the treatment subjects had the opportunity to work on the 
Tutor outside of normally scheduled class times, the majority of their work on the Tutor 
was done during their scheduled laboratory times. The treatment group averaged less than 
three hours on the Tutor outside of normal lecture and laboratory times. Thus, while it is 
possible that any knowledge and skill improvements could be attributed to the fact that the 
tutor group received three hours of work more than the control group, it is unlikely because 
those extra three hours represent less than a three percent increase in time on task. It 
appears more likely that the learning gains were the result of the quality of the students' 
interactions with the Tutor. 

The total number of problems completed by each student before being advanced to 
the next problem type and level depended on their individual performance. Those students 
who displayed stronger problem-solving skills solved fewer problems and ultimately took 
less time to complete the Tutor. Those students who experienced difficulties were cycled 
through additional problems by the computer. Because of the random nature of problem 
selection by the Tutor, the same problem was sometimes accessed by the student a second 
time. Repetition of faults simulates the repetitive nature of actual maintenance activity. 
Technicians see common faults over and over. The repetitive activity required to solve the 
same problems many times enables technicians to develop the mental patterns that relate 
symptoms to faults. 
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Troubleshooting Performance Differences 



Near the end of each semester, all students individually participated in a 
performance task that required them to troubleshoot a faulty aircraft electrical system in 
which four independent faults had been inserted. Observations were conducted and verbal 
protocols were collected as the subjects attempted to locate the inserted faults. The verbal 
protocols were used to validate the researcher observations and verify the problem 
recognition and solution data. 

Ability to Recognize that Faults Exist 

A review of the protocols revealed that the majority of the subjects began their 
problem-finding activity by operating the toggle switches located on the control panel. This 
initial general search typified the system operation checks that are common in aircraft 
maintenance and represented appropriate troubleshooting behavior because no other 
symptomatic information had been provided. 

Them was no significant difference in the ability of the control and tutor groups to 
recognize that faults existed in the electrical system, r(32) = .081, p>.10. All control 
subjects recognized that a fault existed within the beacon and blower motor subsystems. 
Only five control subjects failed to recognize that a fault existed in the gear indicator circuit, 
and only one control subject failed to notice that a fault existed in the instrument light 
circuit. Similarly, all tutor subjects recognized that a fault existed within the beacon, 
blower motor, and instrument light subsystems. Eight tutor subjects failed lo recognize 
that a fault existed in the geai- indicator circuit. The inability of subjects from both groups 
to recognize that a fault existed in the gear indicator subsystem is likely due to the fact that 
the gear indicator does not have a switch on the control panel and the subjects' initial 
problem-finding activity focused on the operation of the control panel switches. Overall, 
the control group was able to recognize ninety-one percent of the faults, while the tutor 
group recognized eighty-nine percent of the faults. 

Ability to Locate Faults 

The most reliable indicator of troubleshooting performance is the realistic 
demonstration of that skill. Data from the observations and verbal protocols were used to 
assess the ability of the subjects to identify the faults in the aircraft electrical system. 
Correct solutions were recorded on the troubleshooting performance task if the exact 
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component-level solution was discovered and verbalized in the protocols. If a subject 
identified a solution that would have removed the fault symptoms, but was not at the 
component level, the performance was judged as correct at the device level. Incorrect 
solutions resulted when the subject identified a component other than the u:ue fault. When 
the subject couid not arrive at a solution or completed the task with a statement such as "I'm 
stuck" or "I can't figure this one out," the solution was recorded as "none." 

In spite of the fact that both groups of subjects were equally able to recognize that 
faults existed within the aircraft electrical system, there was considerable difference in their 
ability to actually locate and identify the faults. The control group subjects solved an 
average of 1.63 problems while the treatment group subjects solved an average of 2.89 
problems at the component level — a seventy-eight percent improvement in troubleshooting 
success over the non-tutor group. As a group, the control subjects solved only twenty-six 
of the sixty- four problems (41%) they faced which represents troubleshooting success on 
less than half of the attempted problems. In contrast, the tutor group subjects solved fifty- 
two of the seventy-two problems they faced for a solution rate of seventy-two percent. The 
distribution of correct problem solutions at the component level for each group are shown 
in Table 2. 

Table 2 

Solution Frequencies for Control and Tutor Group Subjects 



Control Group Solutions Tutor Group Solutions 



# of Solutions 


n 


% 


n 


% 


1 Correct Solution 


9 


56.2% 


1 


5.56% 


2 Correct Solutions 


4 


25% 


5 


27.8% 


3 Correct Solutions 


3 


18.8% 


7 


38.9% 


4 Correct Solutions 


0 


0% 


5 


27.8% 
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As shown in Figure 1 K the ability ot the tutor group to solve more problems at the 
component level than the control group was highly significant, r(32) = -4.290, p<.01. 
Although not statistically significant, nineteen percent of the control group's solutions were 
incorrect while only ten percent of the tutor group's solutions were wrong. The data also 
shows that the control subjects were much more likely to quit working on the problems 
before a solution was found. The control group could not arrive at a solution on twenty- 
two percent of the problems while the tutor group quit on only three percent of the 
problems. This difference was also statistically significant r(32) = 3.192, p<.OL 

A hierarchy of fault difficulty can be assumed from these solution results. The 
blower motor fault, although in the most complex subsystem, elicited the highest level of 
correct solutions for both the control group (81%) and the tutor group (94%). The 
instrument lights fault was successfully solved by eighty-nine percent of the tutor group 
subjects and sixty-three percent of the control subjects. Of particular interest is the beacon 
fault which none of the control subjects could solve at the component level and was only 
solved by thirty-eight percent at the device level. These device level solutions included 
comments such as "replace the beacon" which would result in the replacement of the 
complete device even though the lamp voltage point contact was the only faulty component. 
In contrast, the same problem was correctly solved at the component level by seventy-eight 
percent of the tutor subjects. Eleven of the sixteen control subjects (69%) recognized that 
the gear indicator subsystem contained a fault but only three of them (27%) could locate the 
fault. Ten of the eighteen tutor subjects (56%) reported that a fault existed in the gear 
indicator subsystem and eight of those ten were able to locate the actual fault (80%). 
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Figure 11 
Transfer Task Comparisons 
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Electrical Domain Knowledge Differences 

A domain-referenced test was designed to assess each subject's knowledge of the 
structure, function, and behavior of the electrical system that was simulated in the 
performance task. This declarative knowledge examination was administered in the last 
week of classes after all subjects had completed the course activities and the 
troubleshooting performance task. 

Due to the fact that the tutor group outperformed the control group on the 
troubleshooting performance task, one might suspect that the treatment group had learned 
more about the characteristics of electrical circuits and their components. This, however, 
was not the case. As shown in Table 3, the mean scores on the domain-referenced 
electrical system posttest examination was 19.88 for the control group and 19.82 for the 
tutor group out of a total score of twenty-four. This slight difference in mean scores was 
not statistically significant, r(32) = .073, p>.05. Group comparisons were also made for 
the structural, functional, and behavioral subscales of the domain knowledge examination. 
No significant differences were found between the tutor and control groups for any of the 
domain knowledge subscales. Although the Tutor seems to enhance the troubleshooting 
ability of its users, it does not appear to increase their declarative knowledge of electrical 
system structure, function, and behavior. This finding is consistent with other studies of 
the relationship between electrical domain knowledge and troubleshooting performance 
(Johnson, 1987; Bonaret al., 1986). 

Table 3 

Domain Knowledge Exam Scores for Control and Tutor Group Subjects 



ConU". Group Tutor Group 



Domain Knowledge Scores* 


n 


M 


SD 


n 


M 


SD 




Total Exam Score (24) 


16 


19.88 


2.13 


17 


19.82 


1.94 


.073 


Structural Subscale Score (8) 


16 


6.75 


1.13 


17 


6.47 


1.23 


.679 


Functional Subscale Score (13) 


16 


10.63 


1.31 


17 


10.82 


1.07 


-.477 


Behavioral Subscale Score (3) 


16 


2.38 


.62 


17 


2.47 


1.23 


-.279 


Note: *Number in parentheses 


is total possible 


score. 











"All exam score comparisons are non-significant atp>.05. 
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Cognitive Processing Differences 



Research suggests that expert troubleshooters evaluate problems qualitatively prior 
to taking action (Chi et aL, 1981). Typically this involves making an assessment based on 
available symptoms in order to be able to predict the possible nature of the problem. The 
troubleshooter is then guided by the prediction to select an appropriate strategy for 
identifying the problem. In the process of troubleshooting, the expert constantly monitors 
her or his action to determine whether the initial prediction made about the problem was 
correct. From the feedback received in the monitoring process, the expert could review and 
change strategies accordingly until the problem is finally isolated. This process of thinking 
about one's thinking is referred to as metacognition (Beyer, 1987). 

Metacognition is the focus of a significant portion of the cognitive literature. The 
term metacognition refers to knowing about and controlling one's own thinking processes 
(Brown. 1978), Metacognition includes strategies such as self-monitoring, advance 
planning, self-checking, questioning, summarizing, predicting, generating and evaluating 
alternatives, and evaluating learning. Metacognition appears to be an important factor in 
intelligence, effective learning, and problem-solving ability. Brown (1978) states that "the 
ability to monitor one's own understanding of instructions and messages, whether written 
or spoken, is an essential pre-requisite for all problem solving ability" (p. 83). Bransford 
(1979) extends this idea a step further when he states that "the ability to plan and evaluate 
our own learning strategies seems to be a hallmark of intelligent activity" (p. 244). 

In this study, several cognitive operations were used by both the control and tutor 
subjects. This section examines the cognitive abilities of the subjects to ( 1) develop and 
evaluate hypotheses about potential faults, (2) select appropriate troubleshooting strategies, 
and (3) self-monitor for errors and make appropriate corrections. 

Hypothesis Selection and Evaluation 

The subjects' protocols were examined to identify the number of hypotheses they 
generated. Hypothesis generation is a major phase of the Technical Troubleshooting Model 
and serves as a goal setting process to guide the troubleshooter in the selection of potential 
faults. Hypothesis statements were those comments made by the subjects that suggested a 
potential cause of the problem. For the faults built into the troubleshooting performance 
task, the causes could either be opens, shorts, crossed wires, or burned-out bulbs. Only 
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those statements that suggested a clear goal or hypothesis were considered. Examples of 
hypothesis statements included "I believe I have an open in the circuit," "something is 
switched," and "the problem could be burned bulbs." Some subjects repeated their 
hypothesis statements several times in the course of the troubleshooting process, especially 
when faced with difficulty in resolving the problem. 

There was no significant difference in the ability of the two groups to generate 
plausible hypotheses, r(32) = -.670, p<.05. The control group subjects generated an 
average of 15.3 plausible hypotheses (SD = 5.59) during the transfer task while the tutor 
group generated an average of 16.4 hypotheses (SD = 4.69). This suggests that both 
groups of subjects were equally able to identify potential faults for each problem in the 
transfer task. 

Following the generation of one or more hypotheses that reflect potential faults, the 
troubleshooter must collect and interpret various types of information to determine which 
hypothesis is the actual fault. This information acquisition and interpretation process could 
include making technical checks, examining service manuals, consulting with colleagues, 
or mentally simulating a functional system. During this evaluation process, the 
troubleshooter could make a variety of errors. For example, the troubleshooter could 
incorrectly accept the hypothesis (i.e., by stating that it is the fault when it is not). This 
error would likely result in the troubleshooter replacing functional parts that are not the 
cause of the problem. The troubleshooter could also incorrectly reject the hypothesis (i.e., 
by stating that it is not the fault when it actually is). This error will force the troubleshooter 
to either waste time by considering another hypothesis or to give up on the problem. These 
two errors provide a clear indication that correctly evaluating hypotheses is a critical aspect 
of competent troubleshooting performance. 

While there was no difference between the two groups in their ability to identify 
potential faults, there were significant differences in their ability to correctly evaluate the 
faults. As shown in Figure 12, the tutor group was significantly better able to correctly 
evaluate their hypotheses than the control group, K32) = -2.607, p<.05. As one would 
expect, the tutor group was also less likely to incorrectly evaluate their hypotheses, f(32) = 
2.554, p<.05. Although not statistically significant, the control group was more likely than 
the tutor group to make no decision about the correctness of a hypothesis. These results 
suggest that the experience on the Tutor enhanced the students' abilities to correctly 
evaluate the potential faults they considered. 
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Figure 12 

Correctness of Hypothesis Evaluations 
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Troubleshooting Strategy Selection 

The selection of an appropriate troubleshooting 
competent performance. An important difference was ev 
control and tutor groups. Tutor group members were moj 
in a more in-depth manner prior to selecting a strate; 
incorporated more powerful voltage checks in linea 
dependent upon a single strategy to facilitate the trouble- 

Control group subjects relied on two primary 
linear searches, and (2) visual inspection. Topograpi 
through the subsystem from a reference point, or rever 
visual, voltage, or continuity checks. Additionally, 
subjects (38%) utilized a very inefficient variation of i 
involved checking the continuity of every conductor k 
This stiaicjy was generally supported by continuity ch 
in which it was combined with redundant voltage c 
process. Five of the control group subjects (31%) were 
type independent of the problem type. These subjects ^ 
unable to manage the hypothesis generation and eva 
manner. Only one tutor group member used the ineffie 
time during a single problem scenario and no other : 
behavior. There were also no cases within the tuto 
approach. 

While topographic searches can be effective foi 
suggests that expert troubleshooters evaluate problems 
(Chi et al., 1981). Functional evaluation is an effec 
involves qualitative evaluation of faulty systems (Rasm 
a functional evaluation strategy generally results 
representation and increased efficiency in fault isc 
evaluation strategy, the troubleshooter limits the size 
thoughtful analysis of the circuit without using system ii 
completely supported by a logical evaluation that is basec 
and a review of system diagrams. Some form of functions 



certainly contributes to 
strategy use between the 
) evaluate the symptoms 
oblem space reduction, 
ctivities, and were not 
*ocess. 

pes: (1) topographic or 
OS can be either forward 
.ime manner using either 
•e sixteen control group 
phic search strategy that 
nt in the problem space, 
jgh there were also cases 
n even more inefficient 
ly reliant on one strategy 
tiveiy strategy bound and 
tivities in an appropriate 
luity strategy for a limited 
p members exhibited that 
f a single strategy bound 

ypes of problems, research 
itively prior to taking action 
ubleshooting strategy that 
Jensen, 1974). The use of 
accurate problem space 
Through this functional 
problem space through a 
.ive checks. The strategy is 
the symptomatic conditions 
aluation was used by nine of 
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the sixteen control group subjects (56%) and fourteen of the eighteen tutor group members 
(787r). 

Every member of the tutor group demonstrated an ability to incorporate voltage 
checks within their linear search process. The ability to use this type of information 
acquisition effort has important advantages over the continuity check favored by many 
members of the control group. Voltage measurements enable the troubleshooter in an open 
circuit problem to quickly reduce the problem space and have an indication of circuit 
behavior based on the voltage readings. 

It was noticed that subjects in both groups transferred their experiences gained in 
one problem to the next. This type of behavior appears to be an instance of the Einstellung 
effect (Luchins, 1942). Einstellung is the mental set which induces a person to use an 
action or skill that had just been used successfully in another problem. In other words, 
when an individual has performed one action repeatedly (i.e., practices the action), the 
individual has a tendency or mindset to continue using that action even when it is no longer 
the appropriate behavior. While the Einstellung or history effect is often a negative 
consequence of practice, it was used positively by several subjects. This history effect was 
observed in limited cases in both groups. For example, when a subject found that the 
reason the beacon was not working was due to an open in the device, that same type of 
fault was typically searched for in the next problem. Two subjects in the control group and 
one in the tutor group immediately checked the connections on the beacon circuit relay after 
finding a mis-wired relay in the blower subsystem. One tutor subject used this history 
effect to positive advantage. This subject who found that a jumper wire was helpful in 
obtaining a solution in the Instrument Lights problem, transferred that strategy to the Gear 
Indicator fauh on which he had previously failed to obtain a solution. In this case, the 
history effect resulted in a correct solution. 

The tutor subjects also used a more mature troubleshooting strategy process by 
reverting to trial and error on the Gear Indicator fault when they were unable to collect 
relevant information from other methods. The trial and error efforts were not general 
"shotgun" approaches, however, in that these effor'ts were directed at specific conductors 
or components and not larger segments of the problem space. 
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A clearer picture of the troubleshooting process is developed from a study of the 
strategies and methods employed in that process. Strategies serve as a general framework 
supporting the hypothesis generation and evaluation process through management of 
information acquisition efforts. The ability to select the appropriate strategy is an essential 
element of the troubleshooting process. Weak strategies such as visual inspection can only 
find the most symptomatic faults, and a strict topographic search can easily miss a problem 
that is representative of the behavioral type, particularly additive faults created by marginal 
components. Tutor subjects were generally able to use more powerful strategies and 
change their strategic approaches if necessary. 

Monitoring Errors and Self-Correction 

This metacognitive operation was used to compare the types of errors committed by 
the two groups during information acquisition processes. An important aspect of the 
troubleshooting process is whether the troubleshooter realizes that an error has occurred 
and that self-correction is possible. Significant errors committed by the subjects included 
redundant checks, checks made out of the problem space, and misinterpretations of the 
information they acquired. 

Redundant Checks 

Redundant checks are checks that are not needed because prior information obtained 
by the troubleshooter should have accurately determined the result of the check. The fewer 
the number of redundant checks made, the more efficient the troubleshooter tends to be. 
Two types of redundant checks were commonly made by the subjects: ( 1 ) repeated checks 
and (2) system checks. A repeated check error is committed when the same check is 
executed more than once. This error results when the troubleshooter does not mentally 
keep track of the checks already made. A system check error is committed when the 
troubleshooter executes a check at a level that has been superseded by a previous check. 
An example of a system check error is one in which a subject executes voltage checks at 
points in the circuit before a device even though a previous check had already confirmed the 
existence of voltage at the device. 

The number of redundant check errors made by the two groups differed only 
marginally. The control group averaged 1.04 redundant checks on each problem (59 total 
redundant checks in 57 problem attempts). Similariy, the tutor group averaged 1.03 
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redundant checks on each problem (67 total redundant checks in 65 problem attempts). 
When redundant checks were examined separately, as either repeated or system check 
errors, no significant difference was observed between the two groups. The control group 
made twenty-five repeated errors (42%) and thirty-four system errors (58%), while the 
tutor made twenty-seven repeated errors (40%) and forty system errors (60%). 

There was an apparent difference between the groups in their ability to recover from 
their errors. The control group was able to recover from their errors in only six of the 
seventeen problems where redundant errors occurred (35%), while the tutor group was 
able to recover in eleven of the thirteen problems where redundant errors occurred (85%). 
This finding suggests that the tutor group subjects were both more aware of their errors and 
had the ability to correct them. 

Checks Made Out of the Problem Space 

Checks made out of the problem space are information gathering efforts that should 
not have been done because the acquired information would be of no use to the 
troubleshooter. This type of check would be conducted by a troubleshooter who had made 
a wrong hypothesis selection or may have guessed in the hope that one of the checks would 
reveal something. A good troubleshooter is one who makes few or no errors of this type. 

Observations of the two groups on how well they were able to stay within the 
problem space reveals a marginal difference between them. The control group averaged 
3.1 1 "out of the problem space" checks on each problem ( 177 wrong checks in 57 problem 
attempts) while the tutor group averaged 2.34 "out of the problem space" checks on each 
problem (152 wrong checks in 65 problem attempts). 

Mis in terp reta tions 

These are errors that resulted when the troubleshooter arrived at a wrong solution 
due to misinterpretation of problem information. Misinterpretation errors were classified 
into two types: ( 1) problem management and (2) knowledge deficit. Problem management 
errors are inadvertently or carelessly made even the troubleshooter knows how to perform 
the task. Knowledge deficit errors arise from a subject's lack of knowledge when 
conducting specific checks or interpreting results. The most common problem management 
errors involved forgetting to change the meter function switch and forgetting to switch 
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system power on or off. The most common knowledge deficit errors occurred during 
cither meter reading or interpreting acquired information. This erroneous interpretation of 
the acquired information would lead to a wrong conclusion about the cause of the problem. 

On the whole, the tutor group made fewer misinterpretation errors (30 errors: 77c) 
than the control group (48 errors; 11%). The tutor group subjects also exhibited much 
stronger ability to recover from misinterpretation errors when they did make them than the 
control group. In forty-eight misinterpretations, the control group subjects were able to 
correct their errors only seven times (15%), while the tutor group self-corrected twenty 
times (67%). A closer examination of the data reveals that the misinterpretations did not 
occur uniformly across the subjects and problems. For example, one single subject in the 
control group made twelve misinterpretations on one problem and five on another. When 
the misinterpretation errors are recorded by the number of subjects who committed them in 
order to avoid the distortions that could be caused by a single subject, it was found that the 
tutor subjects still committed fewer errors and had a better recovery rate from their errors. 

Eight tutor group subjects committed problem management errors compared to four 
subjects in the control group. All of the tutor subjects who made problem management 
errors recovered from them and finally solved the problem while only one control group 
subject was able to recover from the error and solve the problem. Out of a total of sixty- 
five problems attempted by the tutor subjects, knowledge deficit errors were committed in 
seven of the problems (1 1%). For the control group, knowledge deficit errors occurred in 
twenty of the fifty-seven problems they attempted (35%). Thus, it appears that the 
enhanced ability of the tutor group subjects to solve the troubleshooting problems may have 
been due, in part, by their greater ability to recover from their mistakes. 

Student Perceptions of the Tutor 

In addition to the collection of comparative performance data, observations of the 
students as they interacted with the Tutor were conducted to obtain formative evaluation 
data. Interviews were also conducted with twelve of the eighteen students in the tutor 
group after they had completed the Tutor. Because the Tutor was in its development suge, 
this formative evaluation was crucial for future improvement of the Technical 
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Troubleshooting Tutor. The following discussion of the evaluation results are organized 
around the key questions that were the focus of the evaluation. 

Student Enjoyment of the Tutor 

After completing the Tutor, ten of the twelve students who were interviewed (83%) 
stated that they enjoyed working with the Tutor while two students (17%) did not. Eleven 
of the twelve students said they would most definitely recommend the Tutor to others. Of 
the two who did not like the Tutor, one stated that he found "the system difficult to 
understand" but nevertheless he felt the pracuce would lead to less time in the field and that 
he would recommend it to others. The second student who did not like the Tutor said it 
was because he "preferred working on actual li\ e projects," When the students were asked 
what they liked about the Tutor, a wide range of factors were cited. Most students stated 
that they liked the graphics, the user-friendliness of the program, and the fact that 
everything they needed for troubleshooting was in front of them on the screen. 

Student Perception of the Difficulty of the Tutor 

The students were asked to indicate the level of difficulty of the problems they 
encountered in the Tutor, Ten (83%) students found the problems challenging while two 
student'^ (17%) said they were not challenging. The majority of the students felt that the 
problems became progressively more challenging as they advanced through the Tutor, One 
student who thought the problems were easy said "the most challenging thing v/as learning 
how the computer wanted you to do things." Those students who thought the problems 
were challenging expressed their views in a variety of ways. One student said, "they were 
pretty challenging, you had to really sit and think." Another student stated that the 
problems "were challenging before I got the hang of it. Later it was possible to pick up the 
solutions by reading the problem information. Towards the end they were harder but I was 
now more systematic," 

It was interesting to observe the different reactions of the students as they attempted 
to locate faults on the Tutor, Difficulty with a problem often evoked some swearing while 
successful solutions brought out displays of happiness that were manifested by laughing 
and talking to oneself, the student working on tlie next computer, or to the supervisor. 



49 



Student Use of the Help Features in the Tutor 

After completing the Tutor, the students were asked how much use ihey made of 
the help, system, and component information modules in the Tutor and how adequate they 
found these to be. Seven students (58%) said they did not use the "HELP" feature built 
into the program at all while four students (33%) reported that they used it a few times. 
Most of the students said that they did not use it because they got all the help they needed 
from the supervisors. A few said that accessing the information would cost "money and 
time" and would, therefore, lower their overall performance. Several students indicated 
that they already knew much about the system and its components and therefore they did 
not need to access that information. The one student who often accessed the supplementary 
information in the Tutor said that he obtained "very helpful clues" from it. 

Changes in Students' Level of Troubleshooting Competence 

.After completing the Tutor, the students were asked whether they felt the Tutor had 
made them better troubleshooters. Nine of the twelve students interviewed indicated that 
they thought it had helped them. Those who feh the Tutor had improved their skills had 
some positive things to say about it. One student said it "helped me organize my planning. 
I was more random before" while another student stated that he could now "narrow down 
[problems] a lot faster." Yet another student said "he would probably use more sensory 
checks now." Another student commented that "It's a very good idea. It helps one sort out 
problems in a ^ ery efficient procedure." 

Changes in the Students' Perceptions of Electrical Systems Courses 

Nine of the twelve interviewed students indicated that they felt the Tutor had 
improved their perception of the electrical systems course. Some of the students reported 
that their prior experience in the prerequisite electrical course was disappointing and they, 
therefore, expected to fare even more poorly in the present course. However, they found 
that the opportunity to solve a large number of simulated problems on the Tutor made them 
feel more positive toward the domain of electricity. One student said that he "was 
previously terrified of electricity. The last course was too abstract." Another student 
captured the view of most of the others when he said "I did not do well in basic electricity 
[but] I now have more confidence." 
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DISCUSSION 



The results of this study show that the students who worked on the Technical 
Troubleshooting Tutor became more effective and efficient troubleshooters than those 
students who did not have the same opponunity. With an average of only five hours on the 
computer, the tutor group showed a seventy-eight percent improvement in actual 
troubleshooting success over the control group. In comparison to the control group, the 
tutor group appeared much more determined to locate the faults and displayed greater 
confidence in their troubleshooting ability. The tutor group also displayed a much higher 
level of competence in meter usage and troubleshooting strategy selection. In cases where 
the subjects made troubleshooting errors, the tutor subjects were better able to correct their 
mistakes and eventually solve the problem. While differences were found in 
troubleshooting performance, no statistical differences were found between the two groups 
in domain knowledge as measured by a domain-referenced examination. These results 
suggest that the Tutor improved the cognitive and metacognitive processes of the tutor 
group but not their declarative knowledge base as measured by conventional means. 

Why did these students become better troubleshooters? This is a difficult question 
to answer because the Tutor is a comprehensive instructional tool that contains numerous 
pedagogical strategies that are supported by educational research. While this study did not 
attempt to identify the specific aspects of the Tutor that made it successful, it may be 
beneficial to speculate about its most effective features to guide future research in this area. 

The primary strength of the Tutor is that it provided students with a structured 
practice environment. Students were confronted with problems that were challenging, yet 
were within their ability range. As students proceeded through the problems, they were 
given corrective feedback and guidance when it was most needed. This structured practice 
environment allowed the students to solve many problems in a short amount of time. In 
effect, the Tutor provided students with the opportunities, experiences, and feedback that 
could be offered by any good instructor if they had an awareness of the essential cognitive 
processes for troubleshooting and the time, equipment, and facility resources available. 

The Tutor problems themselves were also structured in a way that facilitated the use 
of an expert approach to troubleshooting. The students were encouraged to perform like 
experts and were rewarded for doing so. This meant that the students were taught to access 




as much informaiion about the problem as possible, lo use that information to identify 
potential faults, and then to systematically perform checks that would verify the fault. 
Again, the Tutor coached the students through the desired troubleshooting process in the 
same way an effective teacher would guide students through a similar process. 

It is difficult for instructors to turn traditional lab settings into structured practice 
environments. Troubleshooting practice simulators must be developed and sufficient 
numbers of tools and equipment are needed. Students must be kept active in the lab even 
though there are rarely enough work stations for all students to become engaged in actual 
troubleshooting activities. To work around this problem, instructors typically assign 
students a variety of related, but not troubleshooting-specific, tasks that will keep them 
busy in the lab. The use of a computer tutor can easily increase the number of work 
stations in a lab and allows students to practice troubleshooting outside of lab time. Even if 
enough high quality work stations for troubleshooting practice are developed, considerable 
student time will be wasted by the extensive physical manipulations that occur when 
working with real equipment. A strength of the Tutor is its ability to emphasize only the 
cognitive processes of troubleshooting by removing the need for assembly and disassembly 
operations. As a result, the Tutor is able to condense the time frame needed to develop 
troubleshooting expertise. 



Implications for Vocational and Technical Curriculum and Instruction 

This study was designed to identify, refine, and evaluate innovative ways of 
teaching technical troubleshooting. As a result, the findings from this study have 
implications for vocational and technical curriculum development and instructional delivery. 
The following recommendations are based on the results of this study and should be used 
to modify and improve vocational and technical curriculum and instruction: 

• Troubleshooting instruction should place less emphasis on theory and more 
emphasis on troubleshooting skill development. Many people seem to believe that 
the possession of theoretical knowledge is required for competent performance. 
While a theoretical knowledge base may be important, this study and others 
(Johnson, 1987; Lesgold et al, 1986; Rasmussen & Jensen, 1974) show that there 
is little relationship between theoretical knowledge and high-level troubleshooting 
performance. If an instructor's goal is to prepare students who possess strong 
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iroubleshooiing abilities, then the curriculum and the instructional activities must 
emphasize the knowledge and skills that are most needed by troubleshoott^s. This 
means that insiructional designers should primarily focus on system-specific 
content and troubleshooting strategies and emphasize only the theories that are 
fundamental to troubleshooting. 

The processes and skills of troubleshooting should be explicitly taught to students. 
Students cannot be expected to develop troubleshooting skills on their own. Part of 
the difficulty in developing the cognitive skills needed for troubleshooting is that 
those processes occur only in the mind and are, therefore, not directly observable to 
the student. In fact, good thinkers and problem solvers do not even know how 
they think and solve problems because their thought processes have become so 
automated that they occur instinctively (Ericsson & Simon, 1984). Because 
intellectual processes are not directly observable, instructors should explicitly teach 
troubleshooting strategies by explaining not only what the strategy is, but also how, 
when, where, and why the strategy should be employed. 

Students should be provided with extensive opportunities to practice using their 
troubleshooting skills and strategies. Research indicates that practice leads to an 
increase in the speed of task completion and a decrease in the error rate (Phye, 
1986). Through practice, students learn from their successes and failures and they 
begin to develop the ability to process information automatically. Practice is 
essential to the development of automaticity of information processing and skilled 
performance. Practice enables the student to proceed through the stages of skill 
acquisition, from a beginning stage in which information is consciously processed, 
to a stage in which all but unusual information is automatically processed. 
Processing automaticity reduces the load on working memory which allows one to 
think about other task-related things. Extensive amounts of practice also helps 
students "overleam" skills which results in fast and efficient performance. 

Troubleshooting activities should be developed around real problems, situations, 
and contexts. Cognitive research indicates that people learn because of the 
contextual information in the problem situation. The connection between 
environmental cues and problem situations provides students the opportunity to 
process infonnation in a problem-oriented format that enables them to relate the 
available symptoms to a set of potential faults. Students who learn under such 
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siluaied or contextual conditions are more likely to spontaneously use their new 
knowledge in future situations. Consequcndy, careful selection and planning of the 
instrucuonal conte> is of prime importance in instrucuonal design. 

Implications for Future Research and Development 

This study shows the exceptional possibilities for computer-assisted instruction in 
vocational and technical education. Through the development of intelligent tutoring 
systems, vocational and technical education can provide enhanced learning opportunities 
for students. These computer systems have the capacity to guide instruction by realistically 
simulating faulty technical systems and coaching students through the process of 
troubleshooting. These systems appear to be educationally sound and are both time and 
cost-effective. They also have the potential to eliminate many of the problems that 
vocational and technical instructors face when uying to plan and coordinate u-oubleshooting 
activities in laboratories. The following recommendations are made to guide future 
development of the Technical Troubleshooting Tutor and to expand research in this area by 
building on the results from this study: 

• 772^ Tutor needs to be further refined and enhanced to make it more powerful and 
effective. All computer programs need some form of technical support. The Tutor 
is no exception. The current version of the Tutor needs considerable support to 
ensure that the program is running correctly. Refinements of the programming 
code are needed to eliminate any existing "bugs" and to provide enhanced 
processing speed. It is recommended that a complete user's guide be created and 
made available to future users. The guide would describe all of the Tutor features 
anc jnctions, how to use it, how to interpret the performance feedback, and what 
to G when faced with difficulties. Such a document would give instructors the 
confidence to use the Tutor in their courses. Additional faults should be added to 
the Tutor and more digitized images of components and connections would enhance 
the appearance of the program. The potential of multimedia technology should also 
be explored as a way to enhance the Tutor. 

• Provide enough opportunities for the students to become comfortable with the Tutor 
before allowing the Tutor to assess their troubleshooting performance. While the 
students were overwhelmingly satisfied with the briefing they received prior to 
using the Tutor and subsequent assistance they received from the supervisors, it 
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became apparent that many of the students needed help when they began their initial 
use of the Tutor a few days after the briefing. Making the students conscious of 
their troubleshooting performances right from the start seemed to affect their 
willingness to explore various features of the Tutor for fear of being penalized by 
the Tutor. It is recommended that several trial problems be provided where 
performance is not evaluated so the student can explore the features of the Tutor. 

Tlie quality of the feedback and the final performance assessments need to be 
refined and extended. It was observed that some students spent several sessions on 
the Tutor before they actually understood how the Tutor evaluated their 
performance. The feedback the students receive about their troubleshooting 
performance needs to be an accurate evaluation of their performance. If students do 
not understand why they did poorly on a particular aspect of the problem, they will 
become confused and frusu-ated. This is also true if they feel the evaluation is either 
too harsh or does not reflect their true performance. It is also recommended that the 
evaluation process be fully explained before the students begin work on the Tutor. 

Research should be conducted to replicate this study in different fields and at 
different educational levels. This study should be replicated with larger numbers of 
students at different levels of education. The electrical system that is simulated in 
the Tutor is fairly generic in structure and could be used in any electricity and 
electronics courses in high schools, vocational centers, community colleges, and 
industrial training settings. 

Research should be conducted to identify the instructional strategies within the 
Tutor that provide the greatest increase in troubleshooting ability. Because the 
Tutor is a compilation of many instructional strategies, it is not clear which 
su-ategies are most effective. Research that tests individual strategies is needed to 
enhance our understanding of the pedagogy and to guide future changes in 
instructional practice. 

Research should be conducted to test the effectiveness of the Tutor's instructional 
strategies when implemented by human instructors. All of the instructional 
strategies in the Tutor can also be used by human instructors although technical 
instructors need to be trained in the use of these strategies. Research needs to be 
conducted to test their effectiveness in helping students develop the desired 
troubleshooting skills. 
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